Integrating Scientifically Based and Design Experiment Research

This study explored tradeoffs between two research methodologies – scientifically based research and design experiments. We conducted this exploration through a case study of one teacher's four-year implementation of a multimedia learning environment called Astronomy Village®. During the first year of her implementation, she participated in the scientifically based summative evaluation of Astronomy Village. Her students successfully learned complex solar system concepts. In subsequent years, the teacher customized her implementation to meet local needs. An analysis of her design decisions and resulting learning outcomes reveals that the subsequent implementations were more successful than the first-year implementation. Results indicate the importance of tracking long-term implementation. They also reveal tradeoffs between scientifically based and design experiment research. 1 Objectives and Significance The long-term success of any educational program depends on the extent to which teachers can implement the program without direct support from the program developers. A successful scientifically based evaluation should not be the stopping point for a project. It is important to monitor the program after the developers no longer provide direct support to see if the participating teachers can continue to be successful. The summative evaluation of the National Science Foundation-funded Astronomy Village: Investigating the Solar System indicated that students who used the program significantly improved in their understanding of complex solar system concepts. During the summative evaluation teachers were constrained in their implementation based on the quasi-experimental design of the summative evaluation. It was important for comparability to exist across the participating classrooms. After participating in the summative evaluation effort, teachers were free to implement the program in ways that integrated with their local curriculum. Teacher K has implemented the program over four consecutive school years. We have analyzed both the yearly curriculum adjustments as well as the resulting learning outcomes. The goal of this study is to examine the extent to which K was able to adapt the program to her local context in a manner congruent with the goals of the software and to examine the extent to which her adaptations led to increased learning outcomes. 2 Theoretical Framework The design experiment approach, a recent advancement in educational research, is a powerful means to conduct ongoing research and evaluation of educational multimedia (Brown, 1992; Cobb et. al., 2003, McGee & Howard, 1998). Using this approach, researchers work with teachers to implement a program and evaluate the impact of the program on student learning. By reflecting on student performance, teachers and researchers can identify areas of weakness and